Ingress Configuration

Argo CD runs both a gRPC server (used by the CLI), as well as a HTTP/HTTPS server (used by the UI). Both protocols are exposed by the argocd-server service object on the following ports:

  • 443 - gRPC/HTTPS
  • 80 - HTTP (redirects to HTTPS)

There are several ways how Ingress can be configured.

Ambassador

The Ambassador Edge Stack can be used as a Kubernetes ingress controller with automatic TLS termination and routing capabilities for both the CLI and the UI.

The API server should be run with TLS disabled. Edit the argocd-server deployment to add the --insecure flag to the argocd-server command. Given the argocd CLI includes the port number in the request host header, 2 Mappings are required.

Option 1: Mapping CRD for Host-based Routing

  1. apiVersion: getambassador.io/v2
  2. kind: Mapping
  3. metadata:
  4.   name: argocd-server-ui
  5.   namespace: argocd
  6. spec:
  7.   host: argocd.example.com
  8.   prefix: /
  9.   service: argocd-server:443
  10. ---
  11. apiVersion: getambassador.io/v2
  12. kind: Mapping
  13. metadata:
  14.   name: argocd-server-cli
  15.   namespace: argocd
  16. spec:
  17.   host: argocd.example.com:443
  18.   prefix: /
  19.   service: argocd-server:443

Login with the argocd CLI using the extra --grpc-web-root-path flag for gRPC-web.

  1. argocd login <host>:<port> --grpc-web-root-path /

Option 2: Mapping CRD for Path-based Routing

The API server must be configured to be available under a non-root path (e.g. /argo-cd). Edit the argocd-server deployment to add the --rootpath=/argo-cd flag to the argocd-server command.

  1. apiVersion: getambassador.io/v2
  2. kind: Mapping
  3. metadata:
  4.   name: argocd-server
  5.   namespace: argocd
  6. spec:
  7.   prefix: /argo-cd
  8.   rewrite: /argo-cd
  9.   service: argocd-server:443

Login with the argocd CLI using the extra --grpc-web-root-path flag for non-root paths.

  1. argocd login <host>:<port> --grpc-web-root-path /argo-cd

kubernetes/ingress-nginx

Option 1: SSL-Passthrough

Argo CD serves multiple protocols (gRPC/HTTPS) on the same port (443), this provides a challenge when attempting to define a single nginx ingress object and rule for the argocd-service, since the nginx.ingress.kubernetes.io/backend-protocol annotation accepts only a single value for the backend protocol (e.g. HTTP, HTTPS, GRPC, GRPCS).

In order to expose the Argo CD API server with a single ingress rule and hostname, the nginx.ingress.kubernetes.io/ssl-passthrough annotation must be used to passthrough TLS connections and terminate TLS at the Argo CD API server.

  1. apiVersion: extensions/v1beta1
  2. kind: Ingress
  3. metadata:
  4.   name: argocd-server-ingress
  5.   namespace: argocd
  6.   annotations:
  7.     kubernetes.io/ingress.class: nginx
  8.     nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
  9.     nginx.ingress.kubernetes.io/ssl-passthrough: "true"
  10. spec:
  11.   rules:
  12.   - host: argocd.example.com
  13.     http:
  14.       paths:
  15.       - backend:
  16.           serviceName: argocd-server
  17.           servicePort: https

The above rule terminates TLS at the Argo CD API server, which detects the protocol being used, and responds appropriately. Note that the nginx.ingress.kubernetes.io/ssl-passthrough annotation requires that the --enable-ssl-passthrough flag be added to the command line arguments to nginx-ingress-controller.

SSL-Passthrough with cert-manager and Let’s Encrypt

  1. apiVersion: extensions/v1beta1
  2. kind: Ingress
  3. metadata:
  4.   name: argocd-server-ingress
  5.   namespace: argocd
  6.   annotations:
  7.     cert-manager.io/cluster-issuer: letsencrypt-prod
  8.     kubernetes.io/ingress.class: nginx
  9.     kubernetes.io/tls-acme: "true"
  10.     nginx.ingress.kubernetes.io/ssl-passthrough: "true"
  11.     # If you encounter a redirect loop or are getting a 307 response code 
  12.     # then you need to force the nginx ingress to connect to the backend using HTTPS.
  13.     #
  14.     # nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  15. spec:
  16.   rules:
  17.   - host: argocd.example.com
  18.     http:
  19.       paths:
  20.       - backend:
  21.           serviceName: argocd-server
  22.           servicePort: https
  23.         path: /
  24.   tls:
  25.   - hosts:
  26.     - argocd.example.com
  27.     secretName: argocd-secret # do not change, this is provided by Argo CD

Option 2: Multiple Ingress Objects And Hosts

Since ingress-nginx Ingress supports only a single protocol per Ingress object, an alternative way would be to define two Ingress objects. One for HTTP/HTTPS, and the other for gRPC:

HTTP/HTTPS Ingress:

  1. apiVersion: extensions/v1beta1
  2. kind: Ingress
  3. metadata:
  4.   name: argocd-server-http-ingress
  5.   namespace: argocd
  6.   annotations:
  7.     kubernetes.io/ingress.class: "nginx"
  8.     nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
  9.     nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
  10. spec:
  11.   rules:
  12.   - http:
  13.       paths:
  14.       - backend:
  15.           serviceName: argocd-server
  16.           servicePort: http
  17.     host: argocd.example.com
  18.   tls:
  19.   - hosts:
  20.     - argocd.example.com
  21.     secretName: argocd-secret # do not change, this is provided by Argo CD

gRPC Ingress:

  1. apiVersion: extensions/v1beta1
  2. kind: Ingress
  3. metadata:
  4.   name: argocd-server-grpc-ingress
  5.   namespace: argocd
  6.   annotations:
  7.     kubernetes.io/ingress.class: "nginx"
  8.     nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
  9. spec:
  10.   rules:
  11.   - http:
  12.       paths:
  13.       - backend:
  14.           serviceName: argocd-server
  15.           servicePort: https
  16.     host: grpc.argocd.example.com
  17.   tls:
  18.   - hosts:
  19.     - grpc.argocd.example.com
  20.     secretName: argocd-secret # do not change, this is provided by Argo CD

The API server should then be run with TLS disabled. Edit the argocd-server deployment to add the --insecure flag to the argocd-server command:

  1. spec:
  2.   template:
  3.     spec:
  4.       containers:
  5.       - name: argocd-server
  6.         command:
  7.         - /argocd-server
  8.         - --staticassets
  9.         - /shared/app
  10.         - --repo-server
  11.         - argocd-repo-server:8081
  12.         - --insecure

The obvious disadvantage to this approach is that this technique requires two separate hostnames for the API server — one for gRPC and the other for HTTP/HTTPS. However it allows TLS termination to happen at the ingress controller.

Traefik (v2.2)

Traefik can be used as an edge router and provide TLS termination within the same deployment.

It currently has an advantage over NGINX in that it can terminate both TCP and HTTP connections on the same port meaning you do not require multiple hosts or paths.

The API server should be run with TLS disabled. Edit the argocd-server deployment to add the --insecure flag to the argocd-server command.

IngressRoute CRD

  1. apiVersion: traefik.containo.us/v1alpha1
  2. kind: IngressRoute
  3. metadata:
  4.   name: argocd-server
  5.   namespace: argocd
  6. spec:
  7.   entryPoints:
  8.     - websecure
  9.   routes:
  10.     - kind: Rule
  11.       match: Host(`argocd.example.com`)
  12.       priority: 10
  13.       services:
  14.         - name: argocd-server
  15.           port: 80
  16.     - kind: Rule
  17.       match: Host(`argocd.example.com`) && Headers(`Content-Type`, `application/grpc`)
  18.       priority: 11
  19.       services:
  20.         - name: argocd-server
  21.           port: 80
  22.           scheme: h2c
  23.   tls:
  24.     certResolver: default
  25.     options: {}

AWS Application Load Balancers (ALBs) And Classic ELB (HTTP Mode)

ALBs and Classic ELBs don’t fully support HTTP2/gRPC, which is used by the argocd CLI. Thus, when using an AWS load balancer, either Classic ELB in passthrough mode is needed, or NLBs.

  1. $ argocd login <host>:<port> --grpc-web

Authenticating through multiple layers of authenticating reverse proxies

ArgoCD endpoints may be protected by one or more reverse proxies layers, in that case, you can provide additional headers through the argocd CLI --header parameter to authenticate through those layers.

  1. $ argocd login <host>:<port> --header 'x-token1:foo' --header 'x-token2:bar' # can be repeated multiple times
  2. $ argocd login <host>:<port> --header 'x-token1:foo,x-token2:bar' # headers can also be comma separated

ArgoCD Server and UI Root Path (v1.5.3)

ArgoCD server and UI can be configured to be available under a non-root path (e.g. /argo-cd). To do this, add the --rootpath flag into the argocd-server deployment command:

  1. spec:
  2.   template:
  3.     spec:
  4.       name: argocd-server
  5.       containers:
  6.       - command:
  7.         - /argocd-server
  8.         - --staticassets
  9.         - /shared/app
  10.         - --repo-server
  11.         - argocd-repo-server:8081
  12.         - --rootpath
  13.         - /argo-cd

NOTE: The flag --rootpath changes both API Server and UI base URL. Example nginx.conf:

  1. worker_processes 1;
  3. events { worker_connections 1024; }
  5. http {
  7.     sendfile on;
  9.     server {
  10.         listen 443;
  12.         location /argo-cd/ {
  13.             proxy_pass         https://localhost:8080/argo-cd/;
  14.             proxy_redirect     off;
  15.             proxy_set_header   Host $host;
  16.             proxy_set_header   X-Real-IP $remote_addr;
  17.             proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
  18.             proxy_set_header   X-Forwarded-Host $server_name;
  19.             # buffering should be disabled for api/v1/stream/applications to support chunked response
  20.             proxy_buffering off;
  21.         }
  22.     }
  23. }

Flag --grpc-web-root-path is used to provide a non-root path (e.g. /argo-cd)

  1. $ argocd login <host>:<port> --grpc-web-root-path /argo-cd

UI Base Path

If the Argo CD UI is available under a non-root path (e.g. /argo-cd instead of /) then the UI path should be configured in the API server. To configure the UI path add the --basehref flag into the argocd-server deployment command:

  1. spec:
  2.   template:
  3.     spec:
  4.       name: argocd-server
  5.       containers:
  6.       - command:
  7.         - /argocd-server
  8.         - --staticassets
  9.         - /shared/app
  10.         - --repo-server
  11.         - argocd-repo-server:8081
  12.         - --basehref
  13.         - /argo-cd

NOTE: The flag --basehref only changes the UI base URL. The API server will keep using the / path so you need to add a URL rewrite rule to the proxy config. Example nginx.conf with URL rewrite:

  1. worker_processes 1;
  3. events { worker_connections 1024; }
  5. http {
  7.     sendfile on;
  9.     server {
  10.         listen 443;
  12.         location /argo-cd {
  13.             rewrite /argo-cd/(.*) /$1  break;
  14.             proxy_pass         https://localhost:8080;
  15.             proxy_redirect     off;
  16.             proxy_set_header   Host $host;
  17.             proxy_set_header   X-Real-IP $remote_addr;
  18.             proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
  19.             proxy_set_header   X-Forwarded-Host $server_name;
  20.             # buffering should be disabled for api/v1/stream/applications to support chunked response
  21.             proxy_buffering off;
  22.         }
  23.     }
  24. }